The notion of group invariance helps neural networks in recognizing patterns and features under geometric transformations. Indeed, it has been shown that group invariance can largely improve deep learning performances in practice, where such transformations are very common. This research studies affine invariance on continuous-domain convolutional neural networks. Despite other research considering isometric invariance or similarity invariance, we focus on the full structure of affine transforms generated by the generalized linear group $\mathrm{GL}_2(\mathbb{R})$. We introduce a new criterion to assess the similarity of two input signals under affine transformations. Then, unlike conventional methods that involve solving complex optimization problems on the Lie group $G_2$, we analyze the convolution of lifted signals and compute the corresponding integration over $G_2$. In sum, our research could eventually extend the scope of geometrical transformations that practical deep-learning pipelines can handle.

This composition between two finite-sum structures 1 n n i 1 F i ( 1 m m j 1 G j (x)) arises in many machine learning applications such as reinforcement learning [1, 2, 3] and nonlinear embedding [4]. For example, stochastic neighbor embedding (SNE) [4] is a powerful approach to map data from a high dimensional space to a low dimensional space. Let{ z i} n i 1 and { x i} n i 1 denote the representation ofn data points in the high dimensional space and the low dimensional space, respectively. The objective is to pursue a low dimensional representation{ x i} n i 1, such that the distribution in the low dimensional space is as close to the distribution in the high dimensional space as possible. This problem is essentially a composition optimization problem: min x t i p i t log p i t q i t, (2) where p i t exp( ‖ z t z i‖ 2 / 2σ 2 i) j 6 t exp( ‖ z t z j ‖ 2 /2σ 2 i), q i t exp( ‖ x t x i‖ 2) j 6 t exp( ‖ x t x j ‖ 2), lliu8101@uni.sydney.edu.au

We consider the stochastic composition optimization problem proposed in \cite{wang2017stochastic}, which has applications ranging from estimation to statistical and machine learning. We propose the first ADMM-based algorithm named com-SVR-ADMM, and show that com-SVR-ADMM converges linearly for strongly convex and Lipschitz smooth objectives, and has a convergence rate of $O( \log S/S)$, which improves upon the $O(S^{-4/9})$ rate in \cite{wang2016accelerating} when the objective is convex and Lipschitz smooth. Moreover, com-SVR-ADMM possesses a rate of $O(1/\sqrt{S})$ when the objective is convex but without Lipschitz smoothness. We also conduct experiments and show that it outperforms existing algorithms.